gh-149101: Implement PEP 788 by ZeroIntensity · Pull Request #149116 · python/cpython

ZeroIntensity · 2026-04-28T21:41:53Z

Hugo has graciously given me permission to backport this if we don't make the May 5th deadline, but let's try to get this done in time!

I will write a full tutorial and migration guide once this is merged; I want to first make sure that this lands before the beta freeze.

Issue: Implement PEP 788 -- Protecting the C API from Interpreter Finalization #149101

read-the-docs-community · 2026-04-28T21:49:25Z

Documentation build overview

📚 cpython-previews | 🛠️ Build #32483912 | 📁 Comparing fe3d8a1 against main (40dc61a)

🔍 Preview build

25 files changed · ± 25 modified

± Modified

encukou

Thanks for adding these!

I'll send notes for Doc/ now; code review coming up.

encukou · 2026-04-29T09:01:29Z

+   Currently, this function will deallocate *view*, but this may change in
+   the future.
+


Is this relevant for the user? Allocation looks like an implementation detail here.

Not sure -- I added it thinking about debugging (e.g., if you're trying to find a missing PyInterpreterGuard_Close call, you can just look for unfreed PyInterpreterGuard * pointers).

If you're in a debugger already, I don't think you need docs to notice this implementation detail.

encukou · 2026-04-29T09:29:54Z

+   The interpreter referenced by *view* will be implicitly guarded. The
+   guard will be released upon the corresponding :c:func:`PyThreadState_Release`
+   call.
+
+   On success, this function will return the thread state that was previously attached.
+   If no thread state was previously attached, this returns a non-``NULL`` sentinel
+   value. The behavior of whether this function creates a thread state is
+   equivalent to that of :c:func:`PyThreadState_Ensure`.
+
+   To visualize, function is roughly equivalent to the following:
+
+   .. code-block:: c
+
+      PyThreadState *
+      PyThreadState_EnsureFromView(PyInterpreterView *view)
+      {
+            assert(view != NULL);
+            PyInterpreterGuard *guard = PyInterpreterGuard_FromView(view);
+            if (guard == NULL) {
+               return NULL;
+            }
+
+            PyThreadState *tstate = PyThreadState_Ensure(guard);
+            if (tstate == NULL) {
+               PyInterpreterGuard_Close(guard);
+               return NULL;
+            }
+
+            if (tstate->guard == NULL) {
+               tstate->guard = guard;
+            } else {
+               PyInterpreterGuard_Close(guard);
+            }
+
+            return tstate;
+      }


Consider not repeating complex information; it can leave the reader wondering whether it actually is the same.
Only repeat the warning.

Suggested change

The interpreter referenced by *view* will be implicitly guarded. The

guard will be released upon the corresponding :c:func:`PyThreadState_Release`

call.

On success, this function will return the thread state that was previously attached.

If no thread state was previously attached, this returns a non-``NULL`` sentinel

value. The behavior of whether this function creates a thread state is

equivalent to that of :c:func:`PyThreadState_Ensure`.

To visualize, function is roughly equivalent to the following:

.. code-block:: c

PyThreadState *

PyThreadState_EnsureFromView(PyInterpreterView *view)

{

assert(view != NULL);

PyInterpreterGuard *guard = PyInterpreterGuard_FromView(view);

if (guard == NULL) {

return NULL;

}

PyThreadState *tstate = PyThreadState_Ensure(guard);

if (tstate == NULL) {

PyInterpreterGuard_Close(guard);

return NULL;

}

if (tstate->guard == NULL) {

tstate->guard = guard;

} else {

PyInterpreterGuard_Close(guard);

}

return tstate;

}

On success, the interpreter referenced by *view* will be implicitly guarded;

the guard will be released upon the corresponding :c:func:`PyThreadState_Release`

call.

Otherwise, the behavior and return value are the same as for

:c:func:`PyThreadState_Ensure`.

Note that the returned pointer is not necessarily a valid

:c:type:`!PyThreadState` pointer.

encukou · 2026-04-29T09:31:12Z

+
+   This function will decrement an internal counter on the attached thread state. If
+   this counter ever reaches below zero, this function emits a fatal error (via
+   :c:func:`Py_FatalError`).
+
+   If the attached thread state is owned by ``PyThreadState_Ensure``, then the
+   attached thread state will be deallocated and deleted upon the internal counter
+   reaching zero. Otherwise, nothing happens when the counter reaches zero.
+
+   If *tstate* is non-``NULL``, it will be attached upon returning.
+   If *tstate* indicates that no prior thread state was attached, there will be
+   no attached thread state upon returning.
+
+   To visualize, this function is roughly equivalent to the following:
+
+   .. code-block:: c
+
+      void
+      PyThreadState_Release(PyThreadState *old_tstate)
+      {
+            PyThreadState *current_tstate = PyThreadState_Get();
+            assert(old_tstate != NULL);
+            assert(current_tstate != NULL);
+            assert(current_tstate->ensure_counter > 0);
+            if (--current_tstate->ensure_counter > 0) {
+               // There are remaining PyThreadState_Ensure() calls
+               // for this thread state.
+               return;
+            }
+
+            assert(current_tstate->ensure_counter == 0);
+            if (old_tstate == NO_TSTATE_SENTINEL) {
+               // No thread state was attached prior the PyThreadState_Ensure()
+               // call. So, we can just destroy the current thread state and return.
+               assert(current_tstate->owned_by_pythreadstate_ensure);
+               PyThreadState_Clear(current_tstate);
+               PyThreadState_DeleteCurrent();
+               return;
+            }
+
+            if (tstate->guard != NULL) {
+               PyInterpreterGuard_Close(tstate->guard);
+               return;
+            }
+
+            if (tstate->owned_by_pythreadstate_ensure) {
+               // The attached thread state was created by the initial PyThreadState_Ensure()
+               // call. It's our job to destroy it.
+               PyThreadState_Clear(current_tstate);
+               PyThreadState_DeleteCurrent();
+            }
+
+            PyThreadState_Swap(old_tstate);
+      }


Again, leave implementation details to the implementation -- and the PEP :)

Suggested change

This function will decrement an internal counter on the attached thread state. If

this counter ever reaches below zero, this function emits a fatal error (via

:c:func:`Py_FatalError`).

If the attached thread state is owned by ``PyThreadState_Ensure``, then the

attached thread state will be deallocated and deleted upon the internal counter

reaching zero. Otherwise, nothing happens when the counter reaches zero.

If *tstate* is non-``NULL``, it will be attached upon returning.

If *tstate* indicates that no prior thread state was attached, there will be

no attached thread state upon returning.

To visualize, this function is roughly equivalent to the following:

.. code-block:: c

void

PyThreadState_Release(PyThreadState *old_tstate)

{

PyThreadState *current_tstate = PyThreadState_Get();

assert(old_tstate != NULL);

assert(current_tstate != NULL);

assert(current_tstate->ensure_counter > 0);

if (--current_tstate->ensure_counter > 0) {

// There are remaining PyThreadState_Ensure() calls

// for this thread state.

return;

}

assert(current_tstate->ensure_counter == 0);

if (old_tstate == NO_TSTATE_SENTINEL) {

// No thread state was attached prior the PyThreadState_Ensure()

// call. So, we can just destroy the current thread state and return.

assert(current_tstate->owned_by_pythreadstate_ensure);

PyThreadState_Clear(current_tstate);

PyThreadState_DeleteCurrent();

return;

}

if (tstate->guard != NULL) {

PyInterpreterGuard_Close(tstate->guard);

return;

}

if (tstate->owned_by_pythreadstate_ensure) {

// The attached thread state was created by the initial PyThreadState_Ensure()

// call. It's our job to destroy it.

PyThreadState_Clear(current_tstate);

PyThreadState_DeleteCurrent();

}

PyThreadState_Swap(old_tstate);

}

Co-authored-by: Petr Viktorin <encukou@gmail.com>

bedevere-bot · 2026-04-29T16:58:01Z

🤖 New build scheduled with the buildbot fleet by @ZeroIntensity for commit bc78c10 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F149116%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

encukou · 2026-04-30T07:31:27Z

for buildbots: The RHEL8 failures aren't relevant. Refleaks are worrying though.

encukou · 2026-04-30T09:30:04Z

Refleaks are worrying though.

Never mind; main currently leaks (#149179).

encukou

Today's part of the review
(If some comment doesn't make sense, it might be because I didn't read through everything yet. )

encukou · 2026-04-30T09:08:37Z

+[function.PyInterpreterGuard_FromView]
+    added = '3.15'


This is missing from the PEP; I assume that's an oversight. Just update the PEP when you mark it Final, and ask SC to rubber-stamp it.

encukou · 2026-04-30T09:15:54Z

+struct _PyInterpreterGuard {
+    PyInterpreterState *interp;
+};
+
+struct _PyInterpreterView {
+    int64_t id;
+};


These don't need the underscore. (The private header is what makes the members private; if a user included the pycore header they could use PyInterpreterGuard.interp.)

IMO it's better if the struct X name matches the typedef name.

I used the _ prefix not so users don't touch it, but so we (as maintainers) don't accidentally use it. We should use the typedef instead. (Think of this similar to how PyObject is actually struct _object, or how PyInterpreterState is actually struct _is.)

encukou · 2026-04-30T09:22:47Z

+    PyThreadState *isolated_interp_tstate;
+    PyStatus status = Py_NewInterpreterFromConfig(&isolated_interp_tstate, &config);
+    if (PyStatus_Exception(status)) {
+        PyErr_SetString(PyExc_RuntimeError, "interpreter creation failed");


Restore save_tstate here, and in test_thread_state_ensure_nested below.

It might be a better idea to just assert(!PyStatus_Exception(status))? I avoided error checking for some of the other tests. Realistically, there's not any reason this should fail.

encukou · 2026-04-30T09:44:24Z

+    for (int i = 0; i < 10; ++i) {
+        assert(PyThreadState_Get() == save_tstate);
+        PyThreadState_Release(thread_states[i]);
+    }


Shouldn't these be released in opposite order? I guess it doesn't really matter if they're the same, but if that's a detail you're testing here please add a comment.

It's not intentional, good catch!

encukou · 2026-04-30T10:49:53Z

 /* The following places the `_PyRuntime` structure in a location that can be
 * found without any external information. This is meant to ease access to the
 * interpreter state for various runtime debugging tools, but is *not* an
 * officially supported feature */

 /* Suppress deprecation warning for PyBytesObject.ob_shash */
-_Py_COMP_DIAG_PUSH
-_Py_COMP_DIAG_IGNORE_DEPR_DECLS
+_Py_COMP_DIAG_PUSH _Py_COMP_DIAG_IGNORE_DEPR_DECLS


Could you undo the unrelated changes? They make review hard, and in many cases (like this) I disagree with your style choices. If you're fixing style in existing code, make sure it actually conflicts with PEP 7 (and ideally that it also actually bothers you).

Ugh, sorry, this wasn't intentional. I think my editor might have auto-formatted this (and I agree that it's hideous).

encukou · 2026-04-30T11:18:21Z

+        // For debugging purposes, we emit a fatal error if someone
+        // CTRL^C'ed the process.


Should this be removed now?

I think it's nice to have this for users. Otherwise, developers debugging a stuck process will have to pkill every time they mess something up.

encukou · 2026-04-30T11:32:03Z

+        _PyEvent_Notify(&interp->finalization_guards.done);
+        memset(&interp->finalization_guards.done, 0, sizeof(PyEvent));


Is it safe to reset an event like this while something may be waiting on it?

If it is, maybe add a _PyEvent_Reset to pycore_lock, so any future changes are more likely to take this use into account?

We notify the event right before, and the finalization thread can't begin waiting until we release the lock, so nothing should be waiting on it at this point. I think there's a different bug here, though; memset isn't atomic, so the finalization thread might read the zero before it reads the "done" flag from the event.

I'll add a _PyEvent_Reset function that resets it with sequential ordering so this isn't a problem.

encukou · 2026-04-30T11:49:16Z

+// thread state was attached.
+// To do this, we just use the memory address of a global variable and
+// cast it to a PyThreadState *.
+static const int NO_TSTATE_SENTINEL = 0;


Consider adding assert(tstate != NO_TSTATE_SENTINEL) to public functions that take a PyThreadState.

Also consider using an all-NULL PyThreadState struct, for debuggers.

Co-authored-by: Petr Viktorin <encukou@gmail.com>

ZeroIntensity added 6 commits April 28, 2026 15:49

Copy the reference implementation.

43b0798

Merge branch 'main' of https://github.com/python/cpython into pep-788

722dbdf

Document the new APIs.

73cccbb

Add a whatsnew entry.

25e1cf0

Fix stable ABI things.

af1022a

Fix test_embed.

2fb8419

ZeroIntensity requested review from a team, AA-Turner, FFY00, encukou and ericsnowcurrently as code owners April 28, 2026 21:41

bedevere-app Bot added the awaiting core review label Apr 28, 2026

bedevere-app Bot mentioned this pull request Apr 28, 2026

Implement PEP 788 -- Protecting the C API from Interpreter Finalization #149101

Open

ZeroIntensity added 2 commits April 28, 2026 17:43

SIlly news entry.

1096a32

Documentation fixes.

407062f

ZeroIntensity and others added 3 commits April 28, 2026 18:09

Make the sentinel const instead of changing the C analyzer.

a33bdfc

Fix the html IDs job.

c504a9f

Merge branch 'main' into pep-788

f95bfc7

encukou reviewed Apr 29, 2026

View reviewed changes

ZeroIntensity and others added 5 commits April 29, 2026 08:24

Apply suggestions from code review

a00bfbb

Co-authored-by: Petr Viktorin <encukou@gmail.com>

Update Doc/c-api/threads.rst

6ca2499

Co-authored-by: Petr Viktorin <encukou@gmail.com>

Fix lint and remove dead comment.

74259f8

Improve new PyThreadState API docs.

eff9b40

Co-authored-by: Petr Viktorin <encukou@gmail.com>

Fix lint.

bc78c10

ZeroIntensity added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Apr 29, 2026

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Apr 29, 2026

encukou reviewed Apr 30, 2026

View reviewed changes

ZeroIntensity and others added 2 commits April 30, 2026 08:29

Apply suggestions from code review

0a30c25

Co-authored-by: Petr Viktorin <encukou@gmail.com>

Update Doc/c-api/threads.rst

fe3d8a1

Co-authored-by: Petr Viktorin <encukou@gmail.com>

		Currently, this function will deallocate view, but this may change in
		the future.

		// For debugging purposes, we emit a fatal error if someone
		// CTRL^C'ed the process.

		_PyEvent_Notify(&interp->finalization_guards.done);
		memset(&interp->finalization_guards.done, 0, sizeof(PyEvent));

Uh oh!

Conversation

ZeroIntensity commented Apr 28, 2026 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

read-the-docs-community Bot commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Documentation build overview

Uh oh!

encukou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bedevere-bot commented Apr 29, 2026

Uh oh!

encukou commented Apr 30, 2026

Uh oh!

encukou commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

encukou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ZeroIntensity Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ZeroIntensity commented Apr 28, 2026 •

edited by bedevere-app Bot

Loading

read-the-docs-community Bot commented Apr 28, 2026 •

edited

Loading

encukou commented Apr 30, 2026 •

edited

Loading

ZeroIntensity Apr 30, 2026 •

edited

Loading